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ABSTRACT \ * \ . , m # . <i 

In order to . contribute ; to ,the refinement of methods^ 
for assessing the adequacy of program imprementatTon , this study, 
investigated data collection efforts made during the implementation 
of ALERTA , "a bilingual, multicultural preschool curriculum model. 
Comparisons of results" are made with those of . previously "conducted 
studies of program implementation in Segmentary schools,. To aid 
comparison, the discussion - is organized into four sections. The first 
section reviews a representative sample of the literature on 
innovative implementation in the elementary school. The review, 
provides^ a framework for interpreting the ^following description of 
preschool innovation. The second section briefly describes the ALERTA 
program and presents the plan used for assessing the adequacy of its 
implementation. Also included in /this section are samples of the 
descriptions resulting from data-gathering efforts. The third section 
discusses, in relation to existing research in the elementary school, 
strengths and weaknesses of the methods used to assess the adequacy- 
of ALERTA ' s implementation; The fourth, section concludes with 
recommendations offered to Follow Through planners considering 
'assessment of the adequacy of # program implementation. (RH) 
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The continuing debate over 'the -effectiveness and appropriateness of the 
Abt (1977) study of the impact of Follow Through programs has tended to -focuS 
on choice of instruments', methods of' analysis of outcomes and the ftarfow range 
of. outcome^ selected for investigation. This orientation has somewhat obscured 
implications'"^ be drawn from a provocative Abt finding,, that of strong inter-., 
site variations in the*effects of "the different program models'. It'is_pro-' 
bable that such variations were caused by the local context of.. program implement- 
ation which either facilitated or impeded the intervention. House, ofiss, 

•McLean and Walker (1978) point out the importance of context in the inter- 

> \ 
yretation of results, recommending that future evaluations- be sensitive to 

local conditions. , - 

• Accepting that recommendation, NIE is making^ provision for the assessment 
of the adequacy and the dynamics of the implementation of new Follow Through 
models, prior to* the models'' participation in/ assessment of outcomes.^ 
Before sensitivity to the degree and process \>f implementation can ba well ex- 
ercised^ however, 4wo aspects of the work must be addressed: $ 

1. Development or refinement of existing methods for assessing 
the Adequacy of program implementation; and 

2. Identification across settings and program types of ad- 
ditional factors which facilitate or impede the process of^ 
innovation implementation. 

The value of examining both the adequacy and the dynamics of implementation 
is clear. Unless eyaluators can be assured tha t an innovation is in place, they 
cannoj: confidently interpret data on its supposed impact. Less obvious, per- 
haps, is the fact that, assessments' of program implementation must both reflect 
and address the coiap] change 'process: The- -nust -e flexible 

enough to be ui?*d f u .f ">".y * guide program and se 1 decisions, 
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and implementation strategies) and suTTTT^tiX^ly (to inform decisions on pro- 
gram continuance). The identification of factors in local contexts which 
impede or facilitate the process of program implementation is crucial for 
creating a common framework for interpreting data obtained from different y 
settings and/or different program types. This, latter issue is especially # ira- 
portant in the work of tracing children *s progress from preschool through 
the Follow Through experience in the elementary school. 

The purpose of this paper is to contribute to the refinement of methods 
for assessing the adequacy of program implementation through examination of- a 
specif ic instance "oft such data collection in the light, of previously existing 
research. In 1976," the- Administration for Children, Youth and Families (ACYF) 

in the (formed) Department of Health, Education and Welfare,- funded the develop- 

JL ^ • i ~ 

ment pf ALEHTA, a bilingual, multicultural preschool* curriculum model. 4 - Part 

/ * . ^ *" s 

of the coi^jcfitiorfs for funding was that the process of program implementation and « 

the even£kal degree of incrementation at« each of the 'cooperating Head- Start 

sites be carefully and systematically documented. As no further guidance was 

given on the method of that documentation, the curriculum developers had leeway 

to experiment with a variety of techniques** . . 

The extensive pool of project data now being analyzed is yielding 

> 

two types of insights that may be "useful to the Follow Through planning effort. 
On one hand, the interaction of local contexts with the .processes of implement- 1 
ting ALERTA has shown some clear parallels with existing, descriptions' of innova- 
tibn implementation in the elementary school. This is worth notii)g f as Follow Through 



^ALERTA was one of four such models Rinded. The pother three are known as 
Nuevas Fronteras de Aprendizaje, Un Marco 1 Abierto and AMANE^fER. v 

. . . ' ' - .... y .5 ■ V . 
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programs exist in elementary schbcl settings, yet in some cases retain certain 
characteristics- of the preschool programs that gave them birth,. On the 
other hand, the methods of data gathering for the ALERTA program departed in 
significant ways from those generally used in the elementary school change re- 
search, thus opening 'avenues of possibility for the ^identification of some 

i 

additional parameters to the p*rpcess of change. The detail revealed may also 
be found to impact uppn elementary (Follow Through) innovations, "thereby 
having implications for future program design. 

To facilitate comparison of the study of implementation in the two set- 
tings (federally funded preschool vs. municipally funded elementary school), 
this paper is organized into four major sections. The first section reviews 
a representative sample of the literature on innovation implementation 
in the. elementary school, providing an initial framework ^or the interpretation 
of the description of preschool innovation to follow^. The second part briefly 
describes the ALERTA program and presents the plan used for the assessment of 
the adequacy of program implementation. Also included in the second section are 
samples of the descriptions which resulted from trie methods of data gathering 
undertaken. The third ^section of the paper discusses the strengths and weak- 
nesses of the methods used to assess the adequacy of ALERTA* s implementation 
in relation to existing research in the elementafy school; and the fburth 
section concludes with recommendations for. further exploration* in the ^area of 

documentation of ydhange. ' " . 

/ 

Innovation Imp?. •ieht:ation in the Elementary Srhoei 

4 

The work, on innbvatior, implementation in the elementary school *has . 
fallen broadly into four categories: (1) examination of the complexity of change 

(2) discussion of the (often overlooked) importance of the implementation 

. to ... 
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I 

stage to the total innovation effort; (3) identification of factors facilita- ' 
ring or impeding the process of change; and (4) comments on the appropriate- 
ness of the methodology employed in the studies to date. 3*1 the sections 
that follow, a sample of the literature in Bach Qf those areas is reviewed 

to -abstract the major currents pertinent to the present exploration* 

/ j « * 

. - - cj» - 
The Complexity of School Change 

Few who have actively engaged in change efforts would deny that effect- 
ing change is often a highly complex task. Yet, only recently have theorists 
directly addressed the complex nature of school change. - 

Sarason (1971) was bne of the first to recognize the culture of the school 
and the individual as part of a larger * social system. He sees change in terns 

'.of the existing regularities evident £ n the school culture: 

i ■ ■- 

...any attempt to introduce change into the 

school setting requires, among other things, 

changing the existing regularities in some 

% way. The intended outcomes involve changing . . 

an existing regularity, eliminating one or 

more of them, or producing new ones. (p. 63) 

••.In practice, the regularities tend not ♦ 

to be changed and the intended outcomes, 

therefore cannot occur; that is, the more 

^things change i lore t. y remain the same. (p. 86) 

Gcodlad (1975) concurs with Sarason's concept of the School culture arid 
promotes an "ecological" model of education that "goes far beyond schools in 
seeking to embrace people, things and institutions in a systematic, irit^rrelated 
whole." (n. 213) He favi r:s^ the school as the optimum unit for change, em- 
phasizing, that the individual must be viewed within the social context of re- 
gularities within which he/she works. 

9 ' 

Baldridge and Deal* (1975) relate educational change* to organizational 
factors as opposed to taking an individualistic approach. They note that: 
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...educational- change engages all the subsystems 
that together comprise complex educational organi- \ 
zation... These various organizational subsystems 
are related, In -systematic ways. Any subsystem can ' * - 

^pressure another sijlpsystem to*change. (p. 10) 

Thus, whl^le the individual teacher, administrator or school may be ^potential 
unit for change, all related subsystems must be taken into account* for change 
to occur. * \ 

. Kent (1979) points out that instituting needed reforms in their organi- 
zations is one of the tasks administrators are expected- to perform. Like 
the others/ he'emphasizes the complexity of this task. Kent notes^that, in 4 . 
addition to dealing with complex organizational and educational issues, the 
administrator must also "tonstantly keep in mind the political realities of 
their communities and school districts." (p. 239) 

y -a • 

In summary, the culture of the school, .with its interrelated subsystems, 

makes educational change a highly complex matter. Administrators "are often 

^caught between their responsibility for effecting change and their inability 

to change existing -regularities Vi r hin. a complex social' system. r ' 

The Importance of the Iqiple^ent^t ion Stage 

The change literature supports the fact that the implementation >:tage' 
■ » 

is crucial to the success ar failure of an adopted innovation. McLaughlin 
(1976) notes that implementation dominates the innovation process and its 
outcomes in the Rand Change-Agent Study of classroom* organization projects. 
Warren (1976) cautions that as much, if not more, attention must be given 
to the mechanics of * instituting a proposal as ''is given to the purpose for 
its introduction. The importance of implementation is supported- by others, 

> - . 4 * % 

including Gross, Giaquinta and Be1?h§tein (1971)," Sarasor* (1971), Lieberman. 

and Griffin (1976), Mann (1976) , Fullan and Pomfrpt (1977), Loucks (1978) and - 

Herriott and Gross (1979)'. . 



In spite of such support, little is known and understood about actual 
implementation .efforts. Gross; Giacquinta and B.ernstein (1971) point out 
the "paucity of knowi^dge .about the implementation phase of the process of 
planned organizational change- " (p.* 8) And Mann (1976) summarizes the 
gap in the change literature as follows: 

» ** 

With hindsight it is easy to see that designing 
and disseminating change is not, implementing 
change.! What happens inside the school,* at 
the service delivery level,- is absolutely- 
related to our success or failure, yet the 
gap in our knowledge about implementing 
change in schools is formidable. (p. 313) 

Compounding the difficulties" caused by lack of knowledge about innova- 
tion implementation is the fact that impact studies are often conducted before 
innovation- implementation has been fully ascertained. ~- : rr 1978)? in review- 
ing Sussmann's book (1977), calls attention to this pr^ l.em by noting that 

oUjsi ann f s emphasis on the implementation stage and process rather than on 
premature outcome assessment is welcome in a field dominated by an emphasis 
on adoption and measures of success." (p. 203) 

In spite of this need for refinement of. knowledge on innovation imple- , 
mentation, to date the Research and Development Cente^r for Teacher Education 
at the University *df* Texas -(Austin) has been one of the few agencies to ed- 
dress the extent to which individuals have implement!. J adopted innovations. 
Reliable and valid instruments for ^ssessiag levels of use of an innovation 
by individual users (Loucfes, Newlove and Call, e 1976) and for measuring in- 
dividual stages of concern ^"bout innovations (Hali, George and Rutherford, 
1979) have been develpped 'through J:His extensive project;* * 
•It can be concluded that while./fmpi'ementation is a critical stage in - 
lange process, relatively little is known about, the phase; Since 
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this stage is often ignored by those conducting impact studies, one might 
legitimately pose the question: "Why should a norT-iriplemented or only par- 
tially implemented innovation be expected to make a substantial impact?". 

Descriptions of Implementation : 
Factors Facilitating or Impeding 

the Process of Change * . . 

V^ile there i.? general agreement in the literature that additional 
knowledge is needed in the area of innovation implementation, it should be 
noted that- many studies dealing with implementation in the elementary school 
hc.ve been undertaken in the past. Such studies are exemplified by Gross, - 
Giacquinta ^nd Bernstein (1971), Smith and Keith (1971), Bentzen (1974), 
McLaughlin (1976), and Sussmatin (1977). 

» * Certainly there is much of interest to £e gained from past works. 
Gross et. al/ (1971), for example, state four basic obstacles that accounted 
fo'r- the failure of thei* implementation effort. (pp. 196-198) These were: 

1. Teachers 1 lack of clarity about role performance - ■* 
Expected of them. 

2. Teachers 1 lack of- the "skills and knowledge required to 
implement the innovation. 

3. The unavailability of required,. materials and equipment 
• . needed' for imp^einentation. 

4. Organizational arrangements existing prior to and during » 
the innovation's introduction that were incompatible 

with the^ innovation (e. g. , the rigid school schedule). 

These researchers also stress the need for' feedback mechanisms to deal with 
implementation difficulties as they arose. 

Sarason (1971) emphasizes the importance of both programmatic and Ue- 
havioral regularities in implementation efforts. He also stresses the roles 
of both the building principal and the teacher, viewing them tfoth as indivi- 
duals and as participants in the larger school/system culture. 



10 
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JMcLaughlin (1976) notes that in the Rand's Change-Agent Study, "in- 
stitutional receptivity was a necessary but not- a sufficient condition f or^ 
successful iaqjlement^txon." (p. 343) She does, however, cite mutual adapt- 
ation as a conclusive .characteristic of successful implementations of class- 
rood organisation projects exercised in the study, stating: 

Where implementation was successful, and where 
significant, change in participant attitudes, 
" r skills and behavior occurred, implementation 
was characterized by a process of mutual 
, . adaptation in which project go,als and methpds 
vere modified to suit the, needs and interests 
of participants and in which participants 
•changed to meet the requirements of the 
projects — (p. 341) • • 



-Bridge (1976) advocates parental ipvolvement in the implementation of 
change at the school building or district level. Linking theory and re- 
search, Bridge suggests , circumstances under which parents should be involved 

c - 

in decision-making, * the kinds of ^issues that concern parents, and strategies 
for matching parents with suitable tasks. \ 

Survey of the preceding literature and reflection on its implications 
highlight what appear to be four key variables .in the process of change: 

CD institutional receptivity Mid responsiveness; (2) administrative power 



to modify existing regularities in the organizational structure and willing^ 

ness in other ways to participate actively in the steps toward change; (3) 

previous training and present attitudes, knowledge and skill of* staff; and 

(4) .the nature and* extent of parental Involvement. Together they present a ) 

useful initial framework for analyzing data arising from new implementation 

efforts. One c shoul£ be cautious-,- however * not £o view the variables in 

* • * ] 
isolation from the research method^ which have been associated with their 

identification, as those methods have sometimes tended to narrow the field 

* ' . s ' 

of possible questions to be asked» • . 
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Critique of Methodology * 

, Gross et. al (1971) note that many criticisTins can be made of the liter- 
ature cn methodological grounds. These researchers state that rigorous and 
systematic analyses of organizations undergoing change are not generally' made 
- in order to uncover barriers or facilitators to implementation: 

^ Rather, written largely from the prospective of . 

practitioner^ and/or active change agents, 
most explanations are based on highly subjective 
accounts of their experiences during an effort 
to introduce educational change* Typically, no 
supporting evidence is offered aBout conditions 
that are presumed to serve as important factors in- s m 
\ fluencing organizational change. (p. 31) 

Herriott and Gross (1979) criticize past research on the grounds of its limited 

nature. They- note that testing hypotheses ^£out educational innovations is not 

the framework "that has greatest utility for managers of change efforts since 

such studies focus on only a few selected variables. They further state, that 

such analyses: 

...involve only variables relevant to the ' 
problem as it has been delimited by-the 
. investigator. Individuals responsible for 
the management of an educational change 
process, however ,^do "not perform in such 
contrived settings. (p. 39) 

Baldridge and Deal (1975) emphasize the need for practical experience 

as one cf the three things necessary for understanding educational change- 

They state that "either practical implications have not been developed from 

important research studies, or Wimportant research haa unduly influenced the 

development of administrative guideline^and policy. ,f (p. 5) Goodlad (1975) 



concurs, noting that in an ecological model, value judgments should be suspended. 

/ ■ 

until one has a data-base perspective' about the situation at hand. He urges 
that we seek to describe and,' to the best of our abilities understand what it • 
now does 11 'before determining what functions the school should 'fulfill. Goodlad 
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'sees this*as a joint function of "external 11 researcher^ and ''internal 11 in- 

habitants. ! (p. 203) > 

• • \ 

Lieberjnan and Griffin (1976), critiquing the work of Smith and Keith 

(.1971), Grossj et. al. (1971) and Bentzen (1974) , promote an emergent te- 

search method! t;hat would meld the three points of view. ,They notre that: - 

Extending and broadening our methodology and 
clarifying that' extension and breadth will, - •' 

it is believed, soXve some "or the ^problems of 
the lack of comprehensiveness, noted. . . It 
~* must be understood ' that do not propose 
an abandonment of systematic, rigorous, and, 
\ if possible,, elegant research technique * We » 

are arguing for th& formulation of method 
which will, reflect the phenomena being 

studied and, in * effect, capture those . p , 

phenomena fundamentally and accurately, v , / 

(pp. 419-420) . * . > 

. . - • ' ' . . ■ I 

Along- with an emergent method, Griffin (1978) also cites ,the importance 
of formative .evaluation a^arTx)n^going^part of change efforts. This stance 
is supported by Gross et. al (1971) and"lten£^1^9) . Griffin (1978) proposes 
two techniques to meet this need: 

(1) Direct observation by a participant observer 
who regularly and systematically records 
events t of which he or she Is a part . or . by a ^ 
non participant observer who does no ti in- 
teract with the other' persons in the process. 

(2) (Use of) informal interactions and events that occur 
outside formal meetings:., captured by self-reports 

of participants in the forms of logs, diaries... (p. 132) 

In "The Politics" of Training Teachers in Schools", Mann (1976) notes 
that informal evaluation by staff and clientele was an important activity 
in most successful ^pfojects*, and . that those involved, paid attention to it 
at\d. changed because of it. Griffin (1978), stating that evaluation should 

■ /: , s , . - . - 

consider participants* 'time. and energy, also .supports the value of "on the run 
questions and answers. " / 




V 



Williams and Cruikshank ' # . . . Page. 11. 



t Critiques of present methodology strongly support the case study as 
an appropriate ^research method in implementation studies. Gross et. alii 
(1971) cite five reasons for using the case study, including the fact that c 
.data can be collected over time, that research issues can be examined" in 

'»...■. 

natural settings, and that a rapport can be developed and maintained between 4 
the observer and the observed. These , researchers note, of course, that case, 
studies can generate, but do not test theories* Baldridge and Deal (1975) 
also advocate the use of case studies as a means of understanding "whys" of- 

educational innovation and change. They too feel that this method has the 

* jJ » \ ■ • ■ . ■ -/ 

ability to capture the highly complex nature of educational change. Herriott 

and Gross (1979) find the 'case study a highly useful method of examining the 

realities of planned change. They a^so note 'that it provides: 

.♦.the types of data needed to examine 
the dynamics of educational change attempts, 
• a probleip that requires longitudinal data, 
holistic ^perspectives, and an analysis of 
the reciprocal interplay ,amonja^ a complex ;' 
set of variables. (p. 353) 

It should be emphasized ,that while the literature supports the inclusion 

1. . 

o£ qualitative measures in research designs relating to implementation, nowhere /.. 
was it suggested that quantitative measures be abandoned. Rather, such measures 
might be 1 balanced with others of a more qualitative nature in order -to capture * 
the complexities of the change process. . < 

In s^ummary, it can be said that many past^ studies can be criticized for | 
their high subjectiyity or limited methodology. There is a need for an emer- 
gent method, as described by Lieberman, and Griff in, that captures the highly 
complex change process while also offering practical implications for managers 
of change. t This 'should also include formative evaluation mechanisms that can. 

bemused for feed-back purposes during the implementation .phase . Direct 

. * • • o • . 
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observations, as well as logs and diaries were suggested metho'ds of ad~ 

. . . . ■■• » 

dressing thib neecj. The importance of Informal ^valuation ajfd- "on the runH 

questions and,' answers^ was also established. A sfrong case for utilizing 

the crSse" study as a means of describing change efforts was made since it 

has a holistic perspective that can capture the complexity of change oyer 

'. a * / ' •' ■ ■ 

time. ... " / • 

* ! 

It' < ■ . . ■ ' ' 

I > „ i 

-L Assessment of the Adequacy of the Implementation of ALERTA at Four Sites: 
Methods Employed and Samples of Resulting Descriptions 

s ■ . The development of ALERTA was originally commissioned for the purpose of "\ 
presenting Head Start programs, throughout the country with a viable strategy • 
for working with culturally diverse children. Ih the original proposal 
request^ACTF stipulated that the overall orientation to program development*, 
was to be pne of identification of the strengths of minority populations and 
use of those strengths in the furthering of the learning process in young 
children.! It was also expected that community members (parents, Head Start 
staff and other persons with project-related expertise living in the community) * 
►-would be directly involved in the curriculum development process as concep- 
tualizes, materials developers and field testers, working itv a partnership 
with the contractors. - 

. . The curriculum resulting from this interaction consisted of a" nine- 
step process for tailoring early childhood learning experiences to thg* develop- 
mental levels, interests, language proficiency and specific cultural experiences 
brpught by the children and their families to the Head Start center.* Each step 
(or level of program implementation) was set out in manuals for the Head Start 



1 ' '*T ? f . ■ 

This stipulation marked a move away from the deficit orientation which 
Tiad characterized much of the earlier work in Head Start. 
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staff, administrators and parents^ rfnd was supported by' a series of in- 
service contacts (workshops, classroom observations, one^fcpt-one meetings, etc). 
The model included a perspective on child development, a child-centered plan- 
ning technique, strategies ' for fostering continued development in the first 
language and acquisition of a second, and resources for the integtatioji of a 
multicultural perspect£v^4»t^) the total learning experience. J 

The curriculum process and the manuals were developed during the* first 
year of the project (1976-1977). While the program was being field tested ' 

in 1977-1978, ttie need for eventual assessment of the adequacy of program imple- 

' / ■ . ' - - 

mentation was addressed through initial review of research on innovation 
implementation in the elementary school,, and subsequently, through design of 
a v plan and instruments for data collection during tl^e. replication phase to come; 
The third year of program operation (1978-1979) saw substantial revision of the' 
curriculum materials on the basis^oT^^^r^idrtial— field test and the replication 
of the program in two Head Start centers which had npt participated in the 
original material development. It was jover the last five months of the re- 
plication phase that the assessment of the adequacy of program implementation 
was undertaken and a large part of the data was assembled. ACYF decided in 
June of 1979 to extend funding for one more year to support dissemination of 
the curriculum models,* and the program developers were able <£o gather some ad- 
ditional data from other sites that fed into the analyses already begun* 

The synopsis below includes discussion of the derivation of the methods 
used for data collection during the program implementation stage at the seve- 
ral replication sites, sample descriptions arising from the collection, and 
notes on the difficulties experienced in the use of particular techniques. 
Comments on the strengths and weaknesses of the methodology employed iri re- 
lation to that used in earlier research, and recommendations for further work 
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in the design of tools for implementation assessment wil 1 be reserved for the ^ . 

final sections of the paper. , 

*° . * 

Methodology 

At the beginning of/ the second year of the project, the curriculum der . 
velopers could not yet see the total pattern of interaction upon which the 
success of the Implementation would depend. They were aware, however, that 
institutional receptivity (including consistent provision of appropriate 
facilities and materials), administrative commitment to the innovation, and. . 
^ staff and parental attitudes and skills would be likely to prov^ as •signifi- 
cant in the preschool setting»as they had in the elementary school. The de- - 
velopers therefore started their design of protocols and schedules with those 
four factors in mind. In addition, they added to the indicators of the. pro- 
gram's implementation the degree and nature of children's engagement in the ' 
types of activities which the curriculum was meant to promote. 

The' developers were also clear that a variety of techniques would be 
needed to address the different facets of the information to be collected. 
The receptivity of . the institutions involved could in part be assessed through 
a survey of the resources provided. f The gauging of the attitudes, knowledge 
and skills of the administrators, staff and parents, however, as well as the 
tracking of the increasing proficiency of the children in areas particular to 

9 ' ' ... » - 

the program goals, required assessment approaches unique to each circumstance. 

Initially it was "decided that five types of assessment would be employed: 
(1) survey of 'facilities/resources; (2) review .of current documentation (plan - 
books, daily schedules, ^ogs, c child progress folders, etc.); (3) observation ' 
of the children at-work; (4) observation of staff parents and administrators 
at work during both "regular" days and workshops or meetings; and (5) interview 
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of administrators and staff. A sepaf^te set of protocols or schedules wa>s* * 
designed for each purpose. 

Survey of facilities/resources . It was expected that one measure of the 
adequacy of the curriculum's implementation would be reflection of the develop- 
mental, bilingual and multicultural perspectives- of the program* in the . learning 

environment. An environmental survey was designed through an analysis of a 

> • ■ ■ ■ * „ , 

objectifiable curriculum elements (e.g., room arrangements, furnishings, inani- 

* ■ , .** 

pulable materials, design of learning centers, choice of children's literature, 

music, art, etc.) and was conducted a£ three separate times (beginning, .middle 

* *» •***'. ■ . 

and end of "the total assessment period) in each of the- cooperating classrooms. 
■ ' ' ' ■ • #• 

Review of current dQcumentatiori . Incorporation of environmentally/ cul- 

turally appropriate content in both adult-directed and child-initiated acti- 
vities was in part assessed, through periodic inspection of daily and weekly ^< 

planning rorms and observational notes kept on the children. Whether. or not 
" ' ' ' ' ■<,.•'..,♦ 

the curriculum's objectives were being addressed was determined through review . 

, *• 

of the ? iCcrs wider M\re expected to be kept on each child as part of the im- 
plementation urocfess, Us^ _ ;:>f n child-centered' perspective (through provision of 
time for e:^ Id oS^^rv^c^ on.s ind X'-arj planning) andVthe structures for language 
learning were ncrt(?d i.r/. t--: - r sti^g of daily and.weekly schedules.' 

Obser vations of the children . Child observations during the first part of 

the a^s^ssment period were anecdotal, Reflecting the type generally accepted in 

. *»* * * 

- the Child Study literature. Observers from the curriculum developer team were in 
the cooperating classrooms for oner-half day once every two weeks throughout the 
assessment period. . Children to be observed during eacj^^isit. were randomly 
selected within each language proficiency cluster,! so that the full range of 



lgtandard clusters at 'the beginning of a center's use of ALERTA are* monolingual 
Spanish speakers, bilingual Spanish-dominant "speakers, bilingual English dominant 
speakers and monolingual ^English speakers. . (Languages other than Spanish and En- 
glish can be substituted into the pattern*) By the end of the program year,; all of 
q the children are expected to be bilingual to -some degree. 



m*tm 
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change in language u3e could be observed. Each observation was converted to 
a narrative teport and. grouped with others of, given time periods and locations 
for later analysis of patterns and trends in the children's development. It 
was assumed that the children would with increasing frequency display attitudes 
^and behaviors related to the curricular objective^. 

Observations of staff, parents and, administrators . Procedures for obser- 
vation of staff, parents -and administrators at work w£re drawn largely from "* 
ethnographic research techniques.. Specific attitudes and behaviors directly 
related to the tasks of. curriculum implementation were, of course, noted. Con- 
sidered equally important, however,' were the patterns'of interaction observed 
among staff, parents and administrators, which subsequently permitted a "mapping' 1 
of the culture of the center similar to that urged b^ Sarason (1971) and Goodlad e 
(1975). A -particular focus of this effort was th'e identification and tracing 
of any change in the "existing regularities" of the work setting .in response to 
curriculum implementation. The observatidns were done in each cooperating center 
for one-half day once 'every two weeks during the assessment period. 

Interviews of administrators agd staff . Structured interviews of adminis- 
trators and staff were included in the research design for two reasons.' First, 
a method was needed for assessing the knowledge of the curricular principle^ that 
had been acquired during the course, of the implementation. It was of interest 
to^ the curriculum developers to see i-f there' was a close relationship between 
knowledge of the program and skills in its use. 1 Second, such .interviews might 



1 . 

While such a relationship would seem to be without question, Wetzel's (1969) 
work at the University of Arizona (Tucson) shows, it to be a modt point* A 
considerable amount of work remains to be done on definition of the circum- 
stances under which a direct relationship can be observed. 
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serve as a kind of self-report, revealing otherwise, hidden perception* of the 
change process while also providing another type of *the formative evaluation 

. . . \ . ■ * / 

encouraged by Mann (1976) and Griffin (1978). The interviews were conducted 
individually with all of the administrators and staff during the final week 
of the assessment period. \ i.. 

It was expected that vork in each of : the five areas would need to be re- 

• „ J. *~7f 7 

fined as the assessment ojL-inpTeirientation (and. the implementation itself) pro- • 

*■ V- 1 ■•* * ' 

ceded. In fafct, Lieberraan and Griffin's (19,76) "emergent method 11 was beginning" 

to be utilized even though the'developers wiere npt aware of that formulation 

during the' early phase of their work. The particular procedures employed 

ore allowing parameters of the- implementation process not considered in the 

original research p design to show themselves. Those emerging parameters ir 

turn, were dictating change • in seme of the data collection methods, in order 

to capture more of the complexity of the "deepTStructure" processes of change. 

Most immediately, a refinement was seen to be^needed in the observational 
* • " ■" 

technique being used to record the development; of the children. The reciprocal 

■ ■* . ' - \ , ; ■' ,-. . . . 

relationships between the ch'ild i and the classroom environment, child andt child, 
and child and adult were showing some fascinating shifts which needed , to be' more' 
systematically recorded. and analyzed. Consequently, a fc double matrix 'tally sheet 
was designed for rapid noting of the nature and frequency of such interactions. 
The vertical axis of tbe matrix remained constant, refering to the settingor 
class of individuals that evoked the interaction, as follows: 

• 1. On Target Setting Set : * 

a. The child selects the. area 

b. The materials in the area being used in setting specific * 
ways, eg^, in the block area, blocks used .for building 



\ '- 

\ 
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2 ."■ On Target Teacher Set : 

a. The teacher selects the area 

b. The teacher presents the l&terials 
b. " The teacher selects the children ^ 

d. The children will perform the Activity that the teacher 
"specified* ' , 

3. Child Initiated : *• » 

a. The child sole cts the a xea an d/or activity' 



' - f b. The child performs an activity other than what that area 

t designates, e.g., playing in the block area with a table game. 

4. Transition : * 

»' , 

a., Interaction which takes place while waiting for another activity* 

The following ^constitute child/child and child/c^ult .interactions: 

1. Fh>#ical (Tactile) contact 
^/ - 2, Verbal exchange 

3. Non-verbal communication 

a. Physical stance w * ' 

b. Gestures - • * / 

The horizontal axis of the matrix had two variations (1) description of 
classroom area (e.g., family corner, block area, art area, library, etc.)'; and 
(2) reflection of the structure for language development'^ and acquisition (e.g., 
.small group wprk, Spanish as a first language; small group work, Spanish as a 
second language, etc P ). Classrooms implementing ALERTA were expected to show 
a balanced scatter of interactions across the four categories outlined above, ■. 

across areas of the room and in use of the structures for language learning. 

: - , ' ' ~ ■. . * s/ 

v „ 

. The matrix was used ,in place of the earlier* anecdote observations during the 
last six weeks of the assessment period*. o. , , > ? o 

•Another data collection technique which rapidly showed itself to need 
substantial revision was that of the administrative/staff interviews. Subjects 
clearly viewed the interview as a test, even though it w^s-he^d in what was. 
thought to be an easy, conversational manner. 'Consequently, tHey did npt c ' "» ol 



2F : 
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appear to be so relaxed in sharing their knowledge 'as had been hoped they 
would be. During the 1979-1980 program year, the interview was abandoned 
in fatfor of using a list of indicators of program implementation. The in- 
dicators were originally prepared by the curriculum developers, but were sup- 
plemented by the staff, parents and administrators at each site. The list was 
use d in ' workshops By the cooperating groups as a self-evaluati :>f} device* Con- 



versations that S£oj>*£neously arose as each group de^lt'vith the various items 
on the list proved to be far more accurate gauges of the knowledge of individuals 

than the interviews had ever been. H 

• ; > 

Turning to the earlier research^ one should note that Griffin (1978) has 
urged that assessments of program implementation be flexible enough to be used 

, formatively as well, as summatively. The importance of the double purpose of- the . 
assessments to the eventual success^of the implementation of ALERTA shoul<Tn^ \ 
be underestimated. Al^ of the measures' described also acted as channels for fn- 
put from the participants (including the ^children) . Tfiat input was constantly 

•fed back into . the curriculum development or revision process, thus insuring * 
that the crucial mutual adaptation cited by McLaughlin (1976) could occur. 

"The adaptation observed at one of the sites in particular struck the developers 

forcibly. In their Annual Report for- 1979-1980, they noted: J 

The ALERTA -originally' envisioned by* the curriculum developers ,[ 
is not exactly the ALERTA seen lit Center B.- What is ther$ 
is a curriculum that has at its roots trie principles and '* 
theoretical precepts of ALERTA, combined with the center's 
own personality, 'character , and style. We .haVfr seen ,a mutual 
assimilation and adaptation. The ;curriculiim and center are v*. 
one. (p. 20)^ ^ . ' ' « 

The formative, nature of-. the varibus metfioHs of assessment also allowed 

for the data to be analyzed in stages, so that changes could be made in the 

J v. • . ... • , : 



program or- the •work -setting as appropriate to. patterns of interaction revealed. 
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Following are descriptions which arose from the assessment of the adequacy 

\ a" 
of program implement at ibfi, but* whi'ch &lso were used as a basis for. revision , 

/"■.'. " * . ^ • 

of program process, currlcular content or institutional procedure. 

» 

S ample Implementation Descriptions » 

• k> . 

Research in the elementary scfioffl has 'indicated that 0 four factors (insti- 
* * 
tutional receptivity; administrative support and active engagement with the 

innovation; staff attitudes, knowledge and skills; and parental involvement) 

/interact to facilitate or impede program implemeptt&tion^ The same four factors 

were seen to be operative in the preschool setting, *and^the categories proved 

useful* in organizing data for analysis. Within category, ±t soon became evident 

that both complementary aijid different types of insights were coining 'from the va- 

• ° • 

rious methods of data collection used. ... 

' \ 

Institutional receptivity/responsiveness . The analysis of institutional 

interactions in the. implementation process drew upon data gathered through three 

* * * * .4; ■ 

of the measures described above: the* environmental survey, observation of^he 



admipistratprs at work and review of current* documentation at each site* > 

Use of the environmental survey tended to reveal whether or not adminis-. 

trativ^ agencies were being responsive to program needs. At one site, review of 

the survey data led the developers to comment^ in in-hbuse notes:. 

At this point in time,' 26 ci—riculum elementsjare present or 
beginning to be evident in Room 2 l (cri,t^rion for implementation » * 
29 items). More should* have, been obsertfed^ giv&n the v *amount of * - 
time and guidance that the staff has received on preparation 
of the learning environment. Conversation with th^ staff re- . - 

vealed that the teacher has three times requested the educational 
director to purchase the learning materials identified at the last > -' 
workshop, but that as yet, the director has not acted. According to \. 
the director, the problem is that funds for materials purchase have 
.not teen released by the local funding agency. 
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Notes on the observed interaction' between administrators and the local 

funding agency at another site brought to light a rigidity on the part of 

the agency, which impacted heavily on the late of program implementation: 

Great difficulty was experienced orv the part of the center 
directors in obtaining the additional release time requested from 
the Agency in order to carry out the full complement of training 
required for program implementation. Permission for additional 
days requested in August 1978 was not received until March 1979, 
and the directors rightly felt that 9 they could not close the center 
for extra training days in" the absence of that permission. **By 
juggling their existing training' allotment, the directors managed, 
to schedule three days^in a row for preservice, training in Sep- 
tember. As the full preservice workshop is five days long, that 
arrangement put th§\ staff two training days behind at the begin- 
ning of their work. In order to make up for the two training 
days appended Jto their regular one-day inservice in September, 
the directors did not hold inservice training sessions in either 
December or January, the ver^y time when extended contact with 
"the curriculum developers would 0 have been most useful. (Third 
Annual-Report, 1979-1980, pp. 35-36.) > 

.Fortunately for the eventual outcome of the ^implementation process, not 

all of the interactions observed* at this level presented constraints on use 

of the program. In the summaries collected. to prepare .the Annual Report , 

1979-1980 , it was noted that * - K 

One of the moat important reasons why the center . (a third site usjLng 
the program) and~the curriculum became increasingly part of each" 
otljer was thbt the administrators weye able to vork with their Board 
as strong advocates of, the program; Thus, j they were always able to 
secure dates for workshops and observation^ far in advance and did - 
not run the risk of scheduling conflicts which w.ould have under- 
fr mined , the training V T ' 

<r ~c 

Sometimes review of current documentation' at a center Uas equally re- 

vealing of the nature of agency/center interactions. At oiie site r 
. - t - < \&. ■ ' " 

...profiles of the children's progress fouaji. in the "Individual child 
folders gave the impression of all having -been" done at a particular 
point in time (rather than over the span of time expected). The '« 
results were profiles that did not appear representative of the 
^ children's present capabilities. . (Third Annual Report, 1978-1979 , 

pp. 29-30.)^ . • I ~ 
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Investigation of the circumtances under which the profiles \£ere being filled. 

out highlighted another problem. There was no provision for staff planning/ 

a 

evaluation time in the daily schedule set tiy the local funding agency. Thus, 
ongoing recordkeeping posed real difficulties for the staff-.- 

Each of the .descriptions cited above was used "in the assessment to de- 
fine the contexts within which the implementation data should be interpreted. 
They were also used, however, to suggest structural or procedural modifications 
whic}i could enhance, program implementation^ In actual fact, once the difficulties 
were identified, some agencies did move in directions more facilitative to the 
innovation. ' , 

. Adm i nistrative support /engagement . The role of th§ administrators in 
the program implementation was ]4$rtly ? revealed- through the institutional' 'in- 
teractions discussed above. a Further information on the administrators was 
collected through continuing observation of* their wor^, interviews and ob- 
servation of their. use of the list of the implementation irfdica^orsj as well v 
as through review of current program documentation at the center* 

At, ope site it was observed that the English-speaking staff sometimes 

-* ■ ' ! • r 

displayed a negative attitude toward the increasing" use of Spanish in the 

* " . * ■ " ' * • 

classrooms as the program was "being implemented. Th6*director of that Center \' 

* f " . | c ' ' o - 

surmised that what the staff members were actually expressing was discomfort 

\ j .: . 

with their own lack of facility in a second language*., which they perceived as 

■ 

putting , them at. a disadvantage. With the assistance of the curricultnn de- 

* ! 
velopers^the director secured funds" to enable. those staff members to take a 

course in conversational Spanish, if they wished. It is interesting to note 

that all the staff members did take the course, and 'that antagonism toward the 

use Qf Spanish in that Center greatly ' diminished. . 
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Equally potent for the eventual outcome of the program implementation 

were the instances in which administrators showed a tacit lack of support * • 

for nhc inno^/fition. In the course of one, ALERTA implementation, the cutri<- , 

culum develppers were 

♦..somewhat taken aback to experience difficulty in securing 
^ dates for the five days of preservice training. The work- 
shops were scheduled, but repeatedly cancelled without sub- 
stantial explanation by the center administrators. It .was ife 
* until the end of October that ,th£ dates were finally set -Tor . 
thfe sequence... * Another delay of two months in the subsequent ' ' ,[ 
scheduling of the inservice (follow-up) sessions brought the 
ALERTA team to a. frank discussion with„the center directors 
of the probable consequences of their actions for the program ^ 
implementation. (In-house narratives for the preparation of 
the Annual Report, 1979-1980 ) ■ , 



I 



Though discernment of the pattern on non-engagement of the' administrators 

, 4 r. * 

came as a result of the assessment of, program implementation^, the observation 
was able to be used formatively aswell. Open discussion of the difficulty 
led to its partial resolution and some movement on the part of staff toward 
higher levels of program implementation. 

Another sort of insight came from the interviews of administrators. For 
example, at* a center which was experiencing a marked difficulty with certain 

aspects of program implementation , the developers noted: 

'- ! 

Throughout the interview, tjhe director spoke easily and accu- 
rately about the program 1 s theoretical foundations. She fre- 
quently had trouble, however, giving specific examples of ways 
thfe underlying principles could be applied in the daily work 
of .the classrooms. This turnabout was exceedingly interesting 
to the ALERTA developers, as it helped explain the gap that V 
had been observed to exist between the' efforts of the class- 
room staff and the effort of. the director. (Third Annual 
Report, ^978-1979 , p. 32) 

The, role„.of the administrators in implementation became even more* apparent 

when they were leading their staffs in self-evaluation of the adequacy of their 

program implementation, using the list of indicators developed by the -ALERTA 



Williams and Cruikshank _ Page 24. 

r * 

trainers* The attitudes which" the administrators displayed toward their own 

achievements and set-backs, and' their skill at posing alternative ways to 

•reach an agreed upon goal were closely observed by their staff . "Recorded 

descriptions' show a number of instances where staff modelled on the 

behavior of their directors. 

As in the assessment of institutional considerations, review of current 

program documentation proved usefur in discovering administrative attitudes 

toward -the existing regularities of their setting. Early in the program 

assessment, at one site, the developers remarked: 

At present, no individual child folders .are being kept by the 
teaching teams in the classroom to record progress in achievement 
of the goals and objectives. Folders are kept in the front office 
as they always have been, and an assessment on a form previously used 
by Center B is done about a mo^th after the children enter the 
* program. It is not clear what use is subsequently made of the data 
on that form. ° 



As Sarason (1971) would 'have predicted, implementation in the area of 
record keeping did not proceed until the administrators willingly abandoned 
the old structure and stated that the new procedure was to have Individ- 
ual child folders in the classrooms immediately available to the teaching 
teams. Had the implementation assessment not been going: on, it is unlikely 
that this obstacle to record-keeping would have come to the administrator's 
attention. - 

<>■ Staff knowledge/skills/afctitudes . The knowledge, skills and attitudes of 
staff, affected all. aspects of . ^rog^am implementation;- and all the measures 
: used in the assessment. reflected facets ^ of that influence. The environ- . 
mental Survey showed the, staff ^members' ability to.make^heir center exemplify 
the curriculum's b^LSic perspectives. On-gbing observation, of the staff at 



Williams and Cruikshank 



Page 25. 



-work and staff use of the list of program indicators frequently revealed atti- 

A 

tudes of which individual staff members may not have been aware. Review of 
current documentation was an especially helpful way. to assess both the staff's 
conceptualization of the program^; purpose and their management skills. 
Finally, the interviews tended to uncover the schism between knowledge and 
skills which had been suspected from review of previous research. - 

The. four descriptions below give examples of the types of .information 
that came from use of those measures. Once again, indication is given of their 
formative use . 

If ' o 

In the Environmental Survey, Room 2 had 33 elements present 
(criterion for implementation is 29), while Room 1 had 22 
elements at best. In Room 2, all areas were'open and avail- 
able to the children; in Room 1, two of the areas were closed. 
Apother difference lay in the provision of science opportunities % 
(there were several types in Room 2) and, in. use- of the "rotating 
points of interest. 11 Both rooms need additional attention to 
their library corners. Realizing this, the educational director 
has been making a strong effort to seek out and order appropriate 
books. (Third Annual Report , 1978-1979/ 'p. 37. ) A 

■ \ V f 4 * 

In classroom 6, the bilingual teaching assistant was observed 
for 30 minutes as she interacted with the children during, the 
free-choice portion of the day. Almost without exception, the 
assistant used English when she felt it important to "teach! 1 or 
transmit information. She used Spanish only fqr disciplinary* $ 
action or special expressions of approval. D Previously the * c 
assistant had indicated that she felt she* was making full use 
of the ALERTA language strategies. She needs to hear the pattern 
" in her use of the two languages (possibly through audiotaping. a 
sequence) , in .order to understand how much she is departing from 
the strategy.. (Id-house notes for preparation of the Third 
Annual Repor t « 1978-1979.) ~ " 

v * . * * ■ f ' i 

Review of the collected lesson plans for the room showed that the * • 
move toward cultural and environmental specif ieity. -has been gain- 
ing momentum over several months. Use of objectives' for the 
children's learning whichare incorporated into those plans show a ; 
skew^tow^d_^e_.cognitive/language domain — an emphasis cbptrary^ to " ™ .' 
— thai picked up in examination of the individual chilti .profiles. 



This was thought-provoking and made clear the c fact ^thatSnother 
step in staff training must be attention to -each team f s/ coordiha- 
tipn of i^s own recording efforts. (Thir d Annual RetJoZt, 1978-1979 

pp. 30-31.) ' : • :~ ~" • ^ ' - 

28 . ■ 
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- * ^Members of the teaching teams were interviewed in May. Both s 
team members in Room 2 (but especially the assistant teacher) 
were'able to "speak knowledgeably about ALERTA f s language * 
' structure, child groupings, scheduling of the day, lesson 
planning, community resources, role of non-classroom staff 
;and use of culturally specific material. Thus, eight elements 
were covered by 'the team in ways that reflected ALERTA f s intent.* 
(Nine elements covered is considered to' be indicative of 
implementation.) The team in Room 1 was able to speak knowledgeably 
about seven of the elements (the discussion on lesson planning 
did not show, the comprehension sought). This was a surprise.,^ 
gitfen the fact that there appears to 'be^ virtually no translation 
of principles into practice t in Room 1. {Third .Annual, Report 
l»7frr.l979 , p. 38.) 

Most subtle and interesting to the. developers were jjie-^tescript ions 

arising from the tally of the nature and the frequency of the children's 

interactions. While the tally was originally designed to look at the children 

progress, it* was sapn. clear that the deep structures of the staff's use (or 

norf-use)* of the program (priorities, expectations of child behavior, . etc. ) 

were. al^> being traced, as shown in the following examples- 

* When interactions were tallied in each of the classrooms, it 
was found that there was quite an -even scatter over the four 
categories during the* adult-directed portions of the dsy, and 
' £ spread over six classrooms areas during child- initiated 
periods. The area which seems least used is the woodworking 
area. It ,is almost never used by the girls in the classroom- 
Observations over several days showed, the remaining areas to 
be tapped regularly. (Third Annual Report, 1978-1979 , p. 450 ; 

Interaction analyses done (Juring May in Room 2 indicated that 0 
there vas an acceptable spread across the four categories of 
interaction, but that there was a heavy loading *in three areas 
of the room (family corner, water/sand area and table games 
- area). As with Center No. 1, more work needs to be don£ with the . 
teachers on the curriculum possibilities of other areas of the , . 
'room. Room 1 showed limited interactions of all types except } 
dijring snack time, lunch and during the'walk to the park. ^ * • 

. While in the classroom, almost all the children were doing table • 
V games. (Third Annual Report, 1978-1979 , p. 39.) * 
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Shortly before the children* s Christmas break, the number and 
type of interactions were tallied for Room 2. The pattern 
recorded showed consistent .use of the structure for language 
learning, (confirming the earlier observation of the same), and . 
a rather heavy leading toward^use of three areas of the room — 
the family corner, the block. area and the table toys area. 
The loading indicated that the teaching team needed to think ' 
of ways of revitalizing the other areas of their classroom, 
to invite the children' s. more frequent use of them. (Third 
Annual Report, '1978-1979 , p. 29.) 

Parental involvement . Of the four factors interacting in the program im- 
plementation, parental involvement proved most elusive in the assessment. ✓ 
This situatipn was partly the. fault of the'methbd of data collection used 
(as will be discussed below); but it, was also an artifact of the confusion . 
experienced by staff and parents over the nature and purpose of parental in- 
volvement. *What became abundantly clear through the observational technique 
used was that national Head Start policy on parent participation had been 
incompletely assimilated at the local level. Staff and parents often 
"had ^diverging expectations of what constituted involvement; and the attitudes 
of some staff toward parents were ambivalent. In in-house narratives on one 
center's progress,' it was noted : 

It has been pointed out previously that~lT~weak J.ink in Center 
No. 4's overall program lies in its social services staff . 
Because of h staff attitude and lack of follow-through oh -implementa- 
tion-related assignments, the parent program has not been moving 
as. quickly at' the center as ALERTA would wish. > Members of 'the 
ALERTA staff have been assisting the center directors in over- 
coming this difficulty by conducting parent workshops in a manner . 
that provided training. for all three members of the social 
^services staff. It is hopdd tKat. the social service workers will 
be able to pick up and carry on the parent program soon.* (In- 
. house notes for preparation of the Third Annual Report , 1978-1979.) • 

/At dther centers, assessment was much easier because the parental in- 

volvement program was well' developed. Parents volunteered regularly for wort 

in the classrooms and met at least once a month for workshops or policy 

advisement discussions. In these circumstances, use of the participant 
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observation technique was both natural and fruitful, yielding detail important. 

to consic r in use of the program: * 

Discussions with parents who had been participating regularly 
in the monthly meetings showed a beginning ability, to c ascribe 
- £he specifics of the program and a good grasp of the rationale 

behind the bilingual, multicultural approach. All parents 
spoken with seemed pleased with Center No. 3 f s new program,, 
except^ for one motfier (from a.Jewish family) who was upset 
the center had observed Three Kings Day. She felt th^t < 
activities having a religious connotation should jbe avoided 
entirely. This incident gave both the center staff and the 
ALERTA developers much to think about. 

The sample descriptions provided above indicate the flexible nature of. the 

assessment techniques employed* In almost all instances, the data collected 

■ * 

were able to be used for improvement of program use at the "same time as 

9 

th£y were being analysed to determine adequacy of program implementation. 
Thus, a certain economy cf effort ^was achieved and satisfaction given to both 
evaluators and consumers of the program. The assessment was not without its 
difficulties, however. The following section reviews the salient method- 
ological problems experienced. 

Problems Experienced with Methods of * 
. of Assessment Used 

Two types of problems were experienced in the use of the methods described 
those endemic to the nature of the task itself, and those specific to th6-~ ■ 
schedules/protocols, designed for the assessment. * * 

Reflection on th^ir work ^t the end of the implementation assessment led. . 
the. develppers to conclude that Sarason (1971) and G6odl#d (1975) had by no 9 
means exaggerated the complexity of Ihe phenomenon iBp be studied. The .observed 

process of change occurred on many levels and exhibited itself through many 

* • 
facets simultaneously. Even though the methodology had been designed with 

a .certain awarfeness of the' complexity of the change process, it still did not 
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address the complete dynamic of the situation. Analysis of the data revealed 
some, important gaps in what had been meant xo be a total perspective on the 
processes involved in use of the program,- Notable among these was the paucity 
of information on the influence of parental knowledge, skills and support on 
the adequacy of the program's implementation. * 

^Because the complexity of the interaction was fully appreciated only in 
retrospect not all of the needed refinement of ^instrumentation and field 
technique was done before the data were collected. Thus, even those facets 
which were being addressed were not so deeply explored as they might have 
been* 

* 

Another ^general problem experienced was that, due to the need to^meet 
contract requirements, assessment of the implementation's adequacy -had to be 
carried out while the "mutual adaptation" process was still underway at the 
replication sites. For that reason, it was sometimes difficult to sort out 
effects *and determine which were the more. enduring patterns of interaction. 

More specific^ difficulties lay in the construction of, certain of the 
schedules and protocols used. The problem of setting criteria for judging 
adequate implementation of the various aspect's of 4 the program was never 
satisfactorily resolved. Criteria were set for each of the instruments 
(environmental survey, list of indicators, interaction matrix, etc.), but 
they arose more from common sense than from science.^ Again, the implica- 
tions of mutual adaption for criteria* setting were not so closely examined' 
as they should- have been. - 

' r In the open-ended protocols (interview guides, parameters for classroom 
observation, etc.), there were problems in determining whiah, factors observed 
stiould be more- heavily weighted for the data analysis. The criticism of 
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Gross et al. (1971) regarding the intrusion of ^subjectivity in such qualita- 
tive analyses was not met to the complete satisfaction o£ the curriculum de- 
yelopers. ■ * 

Finally, the specific method of review of current program documentation* 
was not sensitive enough to the need for ferreting out the existing regula- 
rities of the setting which were impacting on the implementation process. 
Much more heeded to be done for full use of Sarason's (1971) concept. . . 

Strengths and Weaknesses of the Methods Used ~" 
- * in Relation to Existing Research in the Elementary School 

An analysis of. the methodology used to assess the adequacy xpf ALERTA 1 s 
implementation makes- evident tVo basic strengths. First, the diverse methodo- 
logical approach approximated the "emergent method" advocated by Lieberman 
and Griffin (1976). Qualitative measures utilized in the ALERTA assessment 
included anecdotal records of the children^ development, field studies of 
staff and parents at work, interviews and reviews of current* program documen- 
tation. This "soft data" was coupled with quantitative measures exemplified 
by tallies of the natiirfe and frequency of interactions, and by rating 
of program implementation on, the list of indicators which ultimately replaced 
the periodic^4ptervipws. 

A° second, equally important methodological strength was the use of a case/ 
field study technique. Although modified for purposes of economy of effort, 
the use,, of this approach allowed data to be collected over time in natural 
settings.' The longitudinal effort fostered a rapport between/the observers 
ancf'the observed — a rapport that' might not have been promoted within another, 
methodological framework. The perceived . strengths of the ca.se/field study ' 
approach in the ALERTA assessment paralleled' those described by Gross et. al. 
(1971) for their elementary__school_ja_tudy...J_ J - 
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It was also felt that the case/field study technique offered a vehicle 
for^capturing at least H portion of the highly complex change process. Baldricge 
and Deal (1975), in discussing the use of the case study in the elementary 
school, cite a similar strength. Because the ALERTA program developers 'did 
not undertake the assessment' with preconceived questions regarding the process 
of change, the modified case study (as well as the other qualitative measures 
used) allowed details on factors facilitating or impeding innovation implemen- 
tation to emerge from the data themselves. This -descriptive procedure con- 
curs with Goodlad's (1975) projection of an ecological model for assessment. 

There were also some significant weaknesses in the methodology used. 
Assessment of the adequacy of program implementation was one task among many 
that the developers were undertaking simultaneously to fulfill the contract 
requirements. Consequently, their full attention could not be given to design 
of instruments and data collection. As has been seenT some of the richness 
of detail in the process of change was missed due to this limitation • 

Another weakness was that the quantitative measures used were not valida- 
ted, as had been done with those used in the research of Loucks, Newlove' and 
Hall (1976). Thus, the instruments cannot yet be used for comparisons of data 
across settings and program types. These difficulties in conjunction with 
those summarized in the preceding section of this paper, present a variety of 
possibilities for future research/ in the field of implementation assessment. 

Recommendations to Follow Through Planners 
Considering Assessment of the Adequacy of Program Implementation 

Review of the problems experienced with the ALERTA implementation ftssess- 
taent", and examination of the strengths and weaknesses of the design in relation 1 
to previously existing' research suggest five recommendations for Follow Through 
planners. * ,' • • . 
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Most immediately, program planners should consider the expansion of the 
* usual time allotted for program development to include an additional phase 
specifically for- gathering information on the adequacy of program implementa-* * 
tion; This procedure would reduce, the number of conf ounding variables to be 
sorted out from the implementation data. 

Second, planners should ensure that bodh qualitative an£ quantitative 
measures are included, in the design fojp implementation assessment. An ecolo- 
gical approach is. clearly essential in meeting the* reality of the complexity 
of change. 

Third, the implementation .assessment should include case/field studies 
as part of the qualitative data collection,' with the provision that data „ 
collectors be trained in tf\e essentials of ethnography. 

Fourth, reliable and valid quantitative measures of program implementation 
(such as those originating at the Research and Development Centex* for Teacher % 
Education at the University of Texas) should be utilized across settings -and 
-across program types whenever possible; .Planners should provide .time fqr 
newly* designed quantitative measures ^to be refined, so that the .processes o^f^ > 
establishing^relia^ility and validity can at least be begun in the context of 
a specific implementation assessment. % 

Finally, planners should incorporate time for practice of the techniques 

of assessuent selected,, ao that, situation specific difficulties in; the collec- 

. f * ' ' * 

tion of data can. be met before the implementation assessment phase begins. 

One possible sequence responsive to these recommendations would be: 



Year One Development * of program approach and materials^. Design/ 
choice of measures for assessing the adequacy of program 
implementation. > • * - 

/ ^ • 
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Year Two Field-test of program in selected sites and formative use 
of measures for assessing program implementation, leading' 
. to program revision, refinement of the assessment measures 

^pd practice of all data collection techniques- 

Year Three First seven to nine months: Second field-test of the pro- 
• * . gram at the same sites, to establish mutual adaptation 

a and practice use of refined measures and techniques of 
data collection.- Remaining three to five months: Assess- 
ment«of the adequacy of program implementation* 
• * / 

Year Four Summative evaluation of impact. 

As* noted above, subsequent evaluation of program impact would r^st securely 

r 

on a body of data whidiTcoulcI contribute substantially to its interpretation. 
Thus, a measure of the criticism of previous impact evaluations could be avoided/ 
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